Guide for extracting custom_details fields from NewRelic integration

#1

Hi!

So I’m testing out an alert from NewRelic to Pagerduty using the NewRelic -> PagerDuty service integration.

So I’m trying to use the Event Management -> Service Orchestration Rules for the PagerDuty service, attempting to get PagerDuty to parse the custom_details for NewRelic priority (with a space, unfortunately), if the value is “HIGH” -> set severity to “Warning”, then use the dynamic notifications to alert based on severity-> urgency (warning -> low urgency, default is severity “critical” -> high urgency).

I was able to do with this a Splunk integration, having some trouble with the NewRelic integration.

Can someone spot check my event rules:

event.custom_details['NewRelic priority'] matches part 'HIGH'

given this events API json payload:

{
  "client": "New Relic",
  "client_url": "https://one.newrelic.com",
  "contexts": [
    {
      "href": "https://radar-api.service.newrelic.com/accounts/1401137/issues/<SOME_KEY>?notifier=PAGERDUTY_SERVICE_INTEGRATION&action=",
      "text": "Issue Page",
      "type": "link"
    }
  ],
  "description": "Janus--ethos51prodva6---infra-janus query result is < 3.0 on 'TEST,IGNORE - Janus Response time in POST /janus/v1/sidecar greater than normal'",
  "event_type": "trigger",
  "incident_key": "<REDACTED>",
  "service_key": "<REDACTED>",
  "details": {
    "Alert Condition Names": [
      "TEST,IGNORE - Janus Response time in POST /janus/v1/sidecar greater than normal"
    ],
    "Alert Policy Names": [
      "Janus-Alert-Policy"
    ],
    "Description": [
      "Policy: 'Janus-Alert-Policy'. Condition: 'TEST,IGNORE - Janus Response time in POST /janus/v1/sidecar greater than normal'"
    ],
    "Impacted Entities": [
      "Janus--ethos51prodva6---infra-janus",
      "Janus--ethos51stageva6---infra-janus",
      "Janus--ethos11stagejpn3---infra-janus",
      "Janus--ethos11stageirl1---infra-janus",
      "Janus--ethos02stageva6---infra-janus",
      "Janus--ethos11stageva6---infra-janus",
      "Janus--ethos03prodva6---infra-janus",
      "Janus--ethos03stageva6---infra-janus",
      "Janus--ethos53stageor2---infra-janus",
      "Janus--sensei56stageva6---infra-janus",
      "Janus--ethos02prodva6---infra-janus",
      "Janus--ethos12stageva7---infra-janus",
      "Janus--sensei56prodva6---infra-janus",
      "Janus--ethos12stagenld2---infra-janus",
      "Janus--ethos11stageor2---infra-janus"
    ],
    "IssueURL": "https://radar-api.service.newrelic.com/accounts/1401137/issues/684d494d-b1ea-40e9-8e1f-86d59e69f8ce?notifier=PAGERDUTY_SERVICE_INTEGRATION&action=",
    "NewRelic priority": "HIGH",
    "Runbook": [
      "https://git.corp.adobe.com/adobe-platform/ethos-runbooks/blob/main/janus/sop/troubleshooting.md"
    ],
    "Total Incidents": "15",
    "Workflow Name": "Policy: 3563713 - Janus-Alert-Policy",
    "id": "684d494d-b1ea-40e9-8e1f-86d59e69f8ce",
    "isCorrelated": "false"
  }
}

I believe I’ve interpreted the events notifications path correctly based on this link:
https://developer.pagerduty.com/docs/ZG9jOjM1NTE0MDc0-pcl-overview#paths

and that it seems in the API Event json object, details top-level object seems to what is contained in the custom_details in the event rule.

LMK If any of that’s not correct or I’m missing something, thanks!

#2

The logic looks correct. You mentioned being able to do this with a Splunk integration - was that on the same service or a different one? Something to sanity check is if you are using Service Orchestration Rules or Basic Service Event Rules on the service New Relic is connected to. Within the service’s Event Management settings, just below the button to click to configure the Service Orchestration Rules, it’ll will say what the service is using. Just beneath that will be another button to switch between the two options. It could be a scenario where you are creating Service Orchestration Rules but the service is still sending events through the Basic Event Rules for evaluation.